Making theory-of-mind inferences from human dialogue is a strong indicator of a model's underlying social abilities, which are fundamental for adept AI assistants. However, large language and reasoning models struggle to understand sophisticated social phenomena in transcript data, such as sarcasm and irony. To assess the weaknesses of current models and to identify their solutions, we introduce SocialNLI (SoNLI) -- the first social dialogue inference dataset. SoNLI consists of a collection of dialogue transcripts hand-picked to center complex social nuances like irony and sarcasm, paired with inferences, corresponding likelihood scores, and human-written explanations. We explore social inference analysis as a facet of theory-of-mind, and evaluate LLM and reasoning model theory-of-mind ability through multi-step counterfactual reasoning.

Increasingly, companies are turning to artificial intelligence to understand what people say about their products and services on Twitter or Facebook. The goal is to react more quickly to complainers and perhaps sell more stuff to happy customers. But with sarcasm, there is a big gap between what people say and what they mean. And, because computers tend to take everything literally, they simply don't get the joke. For example, "You look wonderful" can mean two very different things depending on the context and the speaker.